Papers with downstream NLP applications
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Sentence Simplification with Memory-Augmented Neural Networks (N18-2)
Copied to clipboard
| Challenge: | Sentence simplification aims to simplify the content and structure of complex sentences . prior work has focused on monolingual machine translation (MT) and tree-based MT (TBMT). |
| Approach: | They adapt an architecture with augmented memory capacities called Neural Semantic Encoders for sentence simplification. |
| Outcome: | The proposed architecture improves on different datasets and improves human judgments. |
Code-Switching Patterns Can Be an Effective Route to Improve Performance of Downstream NLP Applications: A Case Study of Humour, Sarcasm and Hate Speech Detection (2020.acl-main)
Copied to clipboard
| Challenge: | In this paper, we demonstrate how code-switching patterns can be utilised to improve various downstream NLP applications. |
| Approach: | They propose to use code-switching patterns to improve various downstream NLP applications. |
| Outcome: | The proposed features can improve humour, sarcasm and hate speech detection tasks. |
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sequence-to-sequence models fail to generalize in hierarchy-sensitive manner when performing syntactic transformations. |
| Approach: | They evaluate whether seq2seq models generalize hierarchically on two transformations . they use pre-trained models and their multilingual variants to test their generalization . |
| Outcome: | The proposed models generalize hierarchically on two transformations in English and German. |
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
Evaluating language models for the retrieval and categorization of lexical collocations (2021.eacl-main)
Copied to clipboard
| Challenge: | Lexical collocations are idiosyncratic combinations of two syntactically bound lexical items. |
| Approach: | They perform an exhaustive analysis of current language models for collocation understanding . they first construct a dataset of apparitions of lexical collocations in context . |
| Outcome: | The proposed models perform well in distinguishing light verb constructions, especially if the collocation’s first argument acts as subject, but often fail to distinguish, first, different syntactic structures within the same semantic category, and second, fine-grained semantic categories which restrict the use of small sets of valid collocates for a given base. |
Improving Formality Style Transfer with Context-Aware Rule Injection (2021.acl-long)
Copied to clipboard
| Challenge: | Existing language models pre-trained on large-scale corpora often do not work well for user-generated data where the language styles differ significantly from the mainstream text. |
| Approach: | They propose a method for formality style transfer by injecting multiple rules into an end-to-end BERT-based encoder and decoder model. |
| Outcome: | The proposed method outperforms existing rule-based FST approaches on tweet sentiment analysis tasks. |
A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Word embeddings suffer from unintended demographic biases, a new study shows . word embedders can cause downstream NLP systems to be unfair, the authors argue . |
| Approach: | They propose a metric to evaluate the fairness of word embeddings via the relative negative sentiment associated with demographic identity terms from various protected groups. |
| Outcome: | The proposed metric measures fairness in word embeddings via the relative negative sentiment associated with demographic identity terms from various protected groups. |
On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping (N19-1)
Copied to clipboard
| Challenge: | Sense representations target meaning conflation deficiency but their potential impact has not been investigated in downstream NLP applications. |
| Approach: | They propose to use a reverse dictionary system to address meaning conflation deficiency . they propose to integrate senses into the system to improve semantic understanding . |
| Outcome: | The proposed approach can improve the performance of a downstream NLP application. |
Learning Semantic Correspondences from Noisy Data-text Pairs by Local-to-Global Alignments (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for data-to-text generation use a large-scale training corpus to learn semantic correspondences between structured input data and associated texts. |
| Approach: | They propose a local-to-global alignment framework that uses local and global models to learn semantic correspondences from large-scale datasets. |
| Outcome: | The proposed framework can be generalized to restaurant and computer domains and improve alignment accuracy. |
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Contextualized representations have become the default for downstream NLP applications. |
| Approach: | They propose a method for converting from contextualized representations to static lookup-table embeddings and apply it to 5 popular pretrained models and 9 sets of pretrained weights. |
| Outcome: | The proposed methods show that pooling over many contexts significantly improves representational quality under intrinsic evaluation. |
Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics (2022.coling-1)
Copied to clipboard
| Challenge: | Sentence Textual Similarity (STS) is a classic task that can be applied to downstream NLP applications such as text generation and retrieval. |
| Approach: | They propose a light-weighted Expectation-Correction (EC) formulation for STS computation. |
| Outcome: | The proposed approach is more efficient and scalable than previous approaches. |
Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | A study of multilingual fine-tuning yields better performance on downstream NLP applications . low resource languages such as Oriya and Punjabi are found to be the largest beneficiaries of multi-lingual fine tuning. |
| Approach: | They propose to leverage the relatedness of languages that belong to the same family in NLP models by multilingual fine-tuning. |
| Outcome: | The proposed approach improves performance on downstream NLP tasks by 15% compared to monolingual fine-tuning. |
TabEmb: Joint Semantic-Structure Embedding for Table Annotation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing tables learn by linearizing the 2D table into a 1D token sequence and encoding it with pretrained language models (PLMs) such as BERT, but this leads to limited semantic quality and weaker generalization to unseen or rare values compared to modern LLMs. |
| Approach: | They propose a table annotation module called TabEmb which decouples semantic encoding from structural modeling by creating semantically rich embeddings for each column. |
| Outcome: | Experiments show that TabEmb outperforms baselines on different table annotation tasks. |
Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task. |
| Approach: | They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs. |
| Outcome: | The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings. |
One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Morphologically rich languages are notoriously challenging to process for downstream NLP applications. |
| Approach: | They propose a pretrained model for NLP applications involving the morphologically rich language Sanskrit that outperforms previous models by a considerable margin. |
| Outcome: | The proposed model outperforms tokenized models on established Sanskrit word segmentation tasks and matches the current best lexicon-based model. |
Unifying Discrete and Continuous Representations for Unsupervised Paraphrase Generation (2023.emnlp-main)
Copied to clipboard
Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Jie Fu, Jian Lan, Mei Li, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, Jiancheng Lv
| Challenge: | Existing unsupervised paraphrase generation methods require large-scale, manually annotated paraphrase datasets, which are labor-intensive to build. |
| Approach: | They propose a self-supervised pseudo-data construction method that generates diverse pseudo-paraphrases in distinct surface structures for a given sentence. |
| Outcome: | The proposed method generates diverse pseudo-paraphrases in distinct surface structures for a given sentence. |